System and method for using gaze control to control electronic switches and machinery

ABSTRACT

A system that detects a gaze interaction from a user or animal based on information received from a camera. For example, the user looks directly at the camera or is looking at an angle toward the camera. In response to detecting the gaze interaction, the system identifies an associated action for controlling one or more of an electrical element and a mechanical element. For example, the action may be to control lighting system. The action is then initiated to control the one or more of the electrical element and the mechanical element.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of and claims priority to U.S.patent application Ser. No. 16/029,808, filed on Jul. 9, 2018, entitled“SYSTEM AND METHOD FOR USING GAZE CONTROL TO CONTROL ELECTRONIC SWITCHESAND MACHINERY,” which claims the benefits of and priority, under 35U.S.C. § 119(e), to U.S. Provisional Application Ser. No. 62/659,506,filed on Apr. 18, 2018, entitled “METHOD TO USE HUMAN AND ANIMAL GAZE TOCONTROL ELECTRONIC SWITCHES AND MACHINERY””, which is incorporatedherein by this reference in its entirety. Each of the aforementionedapplications is incorporated herein by reference in their entirety forall that they teach and for all purposes.

FIELD

The present invention is related generally to gaze recognition andspecifically using gaze recognition to control switches and machinery.

BACKGROUND

Currently, there are a couple of ways humans interact withmachines/electrical components. The most common way is by touching or atactile input device, such as a button or touchscreen. Another way tointeract with machines/electrical components that is growing inpopularity is use speech recognition to translate the speech intocommands thereby enabling the machine/electrical component to understandhuman speech. These types of systems can have physical limitations ofwhere a device is incapable of providing voice and or touch interaction.For example, the device may not have an area that can provide tactileinput or does not have digital voice processing capabilities. Forexample, a small sensor may be unable to provide a tactile input orvoice processing capabilities (e.g., supporting a microphone/speaker)due to its size. Alternatively, a device may not be easily reachable dueto distance, thus limiting any tactile input and/or voice access.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a first illustrative system for using gazeto control an electrical and/or mechanical element;

FIG. 2 is a diagram that shows a process for using gaze to control anelectrical and/or mechanical element using gaze quadrants;

FIG. 3 is a flow diagram of a process for using gaze to control anelectrical and/or mechanical element;

FIG. 4 is a flow diagram of a process for using a gaze sequence; and

FIG. 5 is a flow diagram of a process for identifying a gaze quadrant.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a first illustrative system 100 for usinggaze to control electrical and/or mechanical element(s) 104. The firstillustrative system 100 comprises camera(s) 101, a microprocessor 102, amemory 103, and the electrical and/or mechanical element(s) 104.

The camera(s) 101 can be or may include any type of camera 101 that cancapture an image, such as a two dimensional camera, an acoustic camera(a three dimensional camera), an inferred camera, a digital camera, avideo camera, and/or the like. The camera(s) 101 may send a videostream, a picture, and/or the like. The camera(s) 101 may also capturean audio stream associated with the video stream or picture (e.g., froma microphone (not shown)).

The microprocessor 102 can be or may or can include any knownmicroprocessor 102, such as a microcontroller, an application specificmicroprocessor 102, a digital signaling processor, a multi-coreprocessor, and/or the like. The microprocessor 102 interfaces with thecamera(s) 101, the memory 103, and the electrical and/or mechanicalelement(s) 104. The microprocessor 102 may interface with the camera(s)101, the memory 103, and the electrical and/or mechanical element(s) 104either directly (e.g., in the same device and/or on the same circuitboard), via a computer network, via wiring, and/or the like. Forexample, the camera(s) 101 and/or the electrical and/or mechanicalelement(s) 104 may be connected to the microprocessor 102/memory 103 viaa computer network. The microprocessor 102 may comprise multipleprocessors. For example, the microprocessor 102 may comprise amicroprocessor 102 in the camera 101 or a touch screen device along witha central microprocessor 102.

The camera(s) 101, the microprocessor 102, the memory 103, and theelectrical and/or mechanical element(s) 104 may be in the same device.Alternatively, the camera(s) 101 may be external from the microprocessor102/memory 103/electrical and/or mechanical element(s) 104. For example,the camera 101 may be an external camera 101 attached to a computermonitor, an array of cameras 101 embedded in a monitor, a camera 101 ata door, an array of cameras 101 in a room, and/or the like.

The memory 103 can be or may include, a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), aportable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing.

The memory 103 further comprises a gaze application 105. The gazeapplication 105 can detect and manage gaze information received from thecamera(s) 101. The gaze application 105 can initiate different actionsbased on detecting and managing gaze. The gaze application 105 can trackthe gaze or eye of a human and/or an animal. The detection of the gazeinteraction may be used in situations where a user's hands are occupied,where the user 110 does not have hands, where a device is out of armsreach (e.g., a light in a ceiling), where there is no space on thedevice that allows integration of touch interaction, where speechinteraction is not possible, and/or the like.

The gaze application 105 can control a gaze interaction that may be aprimary or a supplementary gaze interaction, which can be used incombination with other types of interactions, such as speechinteractions, touch interactions (e.g., a user 110 touching a touchscreen or button), gesture interactions, and/or the like. For example, aspeech interaction may follow a gaze interaction; in this case, the gazeof a user 110 looking at a device tells the device to turn on amicrophone and then wait/listen for a speech interaction. In anotherembodiment, the gaze application 105 can turn off the microphone when nogaze detected, or in other words, when the user 110 is not looking atthe device.

The gaze application 105 can be used not only to activate a switch or adevice, but also can be used to deactivate the switch or device bymeasuring either the point of gaze (where one is looking) and/or themotion of an eye relative to the head. There are a number of methods formeasuring eye movement that may be employed by the gaze application 105.One variant uses video images from which the eye position is extracted.Other embodiments may use search coils, eye-attached tracking (e.g., aspecial contact lens with an embedded mirror or magnetic field sensor),electric potential measurement (e.g., using electrodes positioned aroundthe eyes of the user 110 to generate an electrooculogram signal), and/orthe like.

In addition, the gaze application 105 can use a video-based eye-tracker.The camera 101 focuses on one or both eyes and records eye movement asthe user 110 looks a particular object. The gaze application 105 can useeye-trackers to track the center of the pupil and infrared/near-infrarednon-collimated light to create corneal reflections (CR). A vectorbetween the pupil center and the conical reflections may be used tocompute the point of regard on the surface of the eye or the gazedirection of the eye. In this embodiment, a simple calibration procedureof the user 110 may be needed before using the gaze application 103.

The gaze application 105 may support various types ofinfrared/near-infrared (also known as active light) eye-trackingtechniques. For example, the gaze application 105 may supportbright-pupil and dark-pupil techniques. Differences between the two maybe based on the location of an illumination source with respect to thecamera 101. If the illumination is coaxial with the optical path, thenthe eye acts as a retroreflector as the light reflects off the retinacreating a bright pupil effect similar to red eye. If the illuminationsource is offset from the optical path, then the pupil appears darkbecause the retroreflection from the retina is directed away from thecamera 101.

Bright-pupil tracking creates greater iris/pupil contrast, thus allowingmore robust eye-tracking with all types of iris pigmentation. Inaddition, bright-pupil tracking greatly reduces interference caused byeyelashes and other obscuring features. Bright-pupil tracking alsoallows tracking in lighting conditions ranging from total darkness tovery bright light. However, bright-pupil techniques are not as effectivefor tracking gaze in outdoors conditions and where extraneous inferredsources interfere with bright-pupil tracking.

In another embodiment, the gaze application 105 may use a process knownas passive light tracking. Passive light tracking uses visible light toilluminate things which may cause distractions to users 110. Anotherchallenge with this passive light tracking is that the contrast of thepupil is less than in active light methods, therefore, the center ofiris is used for calculating the gaze vector instead. This calculationneeds to detect the boundary of the iris and the white sclera (limbustracking). In addition, passive light tracking presents anotherchallenge, such as detecting vertical eye movements due to obstructionof eyelids.

In some embodiments the gaze application 105 may use an eye-trackingdevice. For example, an eye tracking device may be a head-mounteddevice, a chin rest (request the user's head to be stable), an attacheddevice (attached to the user 110) that remotely and automatically trackshead motion. These types of devices typically use a sampling rate of atleast 30 Hz. Although a sampling rate of 50/60 Hz is more common, manyvideo-based eye trackers run at 240, 350 or even 1000/1250 Hz, speeds inorder to capture fixational eye movements and/or correctly measuresaccade dynamics.

Eye movements are typically divided into fixations and saccades (i.e.,when the eye gaze pauses in a certain position and when it moves toanother position respectively). The resulting series of fixations andsaccades is called a scan-path. Smooth pursuit describes the eyefollowing a moving object. Fixational eye movements may include microsaccades: small, involuntary saccades that occur during attemptedfixation. Most information from the eye is made available during afixation or smooth pursuit, but not during a saccade. The central one ortwo degrees of the visual angle (that area of the visual field whichfalls on the fovea) provides the bulk of the visual information; theinput from larger eccentricities (the periphery) has less resolution andlittle to no color, even though contrast and movement is detected betterin peripheral vision. Hence, the locations of fixations or smoothpursuit along a scan-path shows what information loci on the stimuluswas processed during an eye-tracking session. On average, fixations lastfor around 200 milliseconds (ms) during the reading of linguistic text,and 350 ms during the viewing of a scene. Preparing a saccade towards anew goal takes around 200 ms.

Scan-paths are useful for analyzing cognitive intent, interest, andsalience. Other biological factors (some as simple as gender) may affectthe scan-path as well. For example, eye tracking in human-computerinteraction (HCI) typically investigates the scan-path for usabilitypurposes and/or as a method of input in gaze-contingent displays (alsoknown as gaze-based interfaces).

Eye-trackers typically measure the rotation of the eye with respect tosome frame of reference. The frame of reference is usually tied to themeasuring system. Thus, if the measuring system is head-mounted (e.g.,as with Electrooculography (EOG) or a video-based system mounted to ahelmet) then the eye-in-head angles are measured using the frame ofreference. In order to deduce the line of sight in world coordinates,the user's head must be kept in a constant position or its movementsmust be tracked as well. In these embodiments, head direction is addedto eye-in-head direction to determine the gaze direction.

If the gaze application 105 uses a table-mounted device (e.g., as withscleral search coils or table-mounted camera 101 (“remote”) systems),gaze angles are measured directly in world coordinates. Typically, inthese situations head movements are prohibited. For example, the headposition is fixed using a bite bar or a forehead support. In thisembodiment, a head-centered reference frame is identical to aworld-centered reference frame. Or colloquially, the eye-in-headposition directly determines the gaze direction.

In one embodiment, the gaze application 105 may be a distributed gazeapplication. The gaze application 105 may be in multiple devices thatcommunicate via a computer network, such as the Internet. For example,the gaze application 105 may be in a distributed access system thatmanages door access in multiple buildings.

The electrical and/or mechanical elements 104 can be or may include avariety of elements, such as, a computer system, an electronic switch,an electronic control (e.g., a volume control), a relay, a system thatsends electronic messages, an alarm system, a door lock, a disk space(e.g., grant access to the disk space), a dog/cat feeder, a vehiclelock, a garage door opener, a vehicle starter, a lighting system, anindividual light, an electrode (e.g., for moving a muscle), a listeningdevice (e.g., Amazon's Alexa®), a printer, a scanner, a computer, alaptop, a note pad, a heating/air system, a sprinkler system, anindividual sprinkler, a temperature gauge, a card scanner, a biometricscanner, a sensor, a camera 101, and/or the like.

The first illustrative system 100 also shows a user 110. The user 110may be any person that has eyes that can be used to detect gaze. In FIG.1, the user 110 is shown to be in view of the camera(s) 101. Althoughnot shown in FIG. 1, an animal may also be within view of the camera(s)101 instead of the user 110 (or along with the user 110).

FIG. 2 is a diagram 200 that shows a process for using gaze to controlan electrical and/or mechanical element 104 using gaze quadrants201A-201N (also identified by the numbers 1-9). The diagram 200comprises the gaze quadrants 201A-201N and the user 110 (a person). Thegaze quadrants 201A-201N are used to represent where a user 110 isgazing (view angles from the center gaze). In FIG. 2, there are nineillustrative gaze quadrants 201A-201N. However, in other embodiments,there may be more or less gaze quadrants 201. For example, there mayonly be three gaze quadrants (a top gaze quadrant (the area of gazequadrants 201A-201C), a center gaze quadrant (the area of gaze quadrants201D-201F), and a bottom gaze quadrant (the area of gaze quadrants201G-201N).

If the user 110 is looking directly at the camera 101 as shown in step210A, the user 110 is gazing at gaze quadrant 201E (the center gaze). Ifthe same user 110 then looks up as shown in step 210B, the user 110 isgazing at gaze quadrant 201B. In this example, the gaze angle 211changes a number of degrees, which is detected by the gaze application105 (e.g., using one or more of the processes described above).Likewise, when the user 110 gazes at the gaze quadrant 201C (or anyother gaze quadrant 201), a different gaze angle (e.g., a left gazeangle and an up gaze angle in this example) is determined (e.g., basedon a difference from the center gaze) to identify that the user 110 islooking at the gaze quadrant 201C. The gaze of the user 110 may becoupled with a time period. For example, the user 110 may have to lookat the gaze quadrant 201 for a specific time period (e.g., 2 seconds).

When the user 110 gazes at one of the gaze quadrants 201 (e.g., for thespecific time period), there is an action associated with the gazequadrant 201. Each of the gaze quadrants 201A-201N can have a specificaction associated with the gaze quadrant 201. For example, when the user110 gazes at the gaze quadrant 210E in step 210A, the action may be toask the user 110 if he wants to place an order for a product from aspecific service provider (e.g., Amazon®). The user 110 then can thensay yes or no. If the user 110 says yes, the user 110 can then proceedand place the order. If the user 110 gazes that the gaze quadrant 201Bas shown in step 210B, the action may be to ask the user 110 if he wantsto place an order using Google®. The user 110 can then say yes or no.Likewise, different actions may occur when the user 110 gazes at thevarious other gaze quadrants 201.

The gaze quadrants 201A-201N may be for associated actions where thegaze quadrants 201A-201N change based on different contexts of theassociated actions. For example, gaze quadrant 201E (the center gaze)may be used to turn on a radio, which then changes the context of thegaze quadrants 201A-201N. For example, the gaze quadrants 201A-201D and201F-201N may then be used to the select particular radio stations,particular playlists, and/or albums while the gaze quadrant 201E (thecenter gaze) is used to turn off the radio.

Alternatively, the gaze quadrants 201A-201N may be used for completelyunrelated actions. For example, gaze quadrant 201A may be to turn on thelights and gaze quadrant 201B may be used to place a voice call to aparticular user 110.

In one embodiment, as the user 110 changes the gaze angle from one gazequadrant 201 to another gaze quadrant 201, the user 110 may be providedwith an audible and/or visible indication of the action associated withthe gaze quadrant 201. For example, using the radio example from abovewhere the gaze quadrants 201A-201D and 201F-201N represent individualradio stations, when the user 110 is gazing in the gaze quadrant 201A,the radio says “country 107.9, select?” The user 110 could then say“select” to select the radio station or change the gaze angle to adifferent gaze quadrant 201. For example, when the user 110 changes thegaze angle to gaze at gaze quadrant 201B, the radio says “rock 97.5,select?” The user 110 could then say “select” to select the radiostation. When the user 110 gazes at the gaze quadrant 201E, the radiosays “turn off the radio?” The user 110 could the say “yes” and turn offthe radio.

In one embodiment, the gaze quadrants 201A-201N may be used by anotherparty (a party who is not being viewed by the camera 101). For example,the gaze application 105 may be part of an alarm system that is used todetect unauthorized access to a building or facility (a restrictedarea). When an unauthorized user 110 enters the restricted area and theuser 110 does not gaze directly at the camera 101, a silent alarm isgenerated (e.g., a security guard's screen or phone flashes/vibrates).When the unauthorized user 110 looks directly at the camera 101, anaudible alarm sounds. The alarm system may also use other types ofinput, such as, facial recognition, biometrics, and/or the like.

In one embodiment, the gaze quadrants 201A-201N can be used to detect agaze sequence 220 by the user 110. For example, as shown in FIG. 2, thegaze quadrants 201A-201N can be used to detect the gaze sequence 220.The gaze sequence 220 is where the user 110 gazes using the gazesequence 1->4->5->8->9 (represented by the numbers at the top of thegaze quadrants 201A-201N) to initiate an action. For example, the gazesequence 220 may be used to authenticate a user 110 (e.g., where thereis no display or keyboard). The gaze quadrants 201A-201N may represent anon-displayed keypad (representing the numbers 1-9). The user 110 canselect the gaze quadrant 201 by looking at the gaze quadrate 201 (anangle from center) for a time period. This may be coupled with a voiceinput. For example, the user 110 may say “select” when looking at thegaze quadrant 201A or “done” when entering the gaze sequence 220.

In one embodiment, the gaze quadrants 201A-201N are actually printed ona wall (or displayed on a device). For example, the gaze quadrants201A-201N could be printed on a wall to look similar to FIG. 200(without any text or numbers) where the camera 101 is located in thecenter of gaze quadrant 201E. Alternatively, the gaze quadrants201A-201N could have text or numbers (e.g., 1-9 as shown in FIG. 2). Inthis example, the gaze sequence 220 represents the PersonalIdentification Number (PIN) of 1->4->5->8->9. This type ofauthentication could be coupled with audible sounds (e.g., “select”) asthe gaze moves from one gaze quadrant 201 to another gaze quadrant 201.In this embodiment, the audible sound would typically not state thenumbers because another person could be listening and hear the PIN. Inthis embodiment, the gaze quadrants 201A-201N act like keypad withoutactually using a mechanical keypad.

In one embodiment, the gaze sequence 220 may be coupled with facialrecognition. For example, the user's face is compared to a stored faceprint of the user 110. Both the face print and the gaze sequence 220have to match. In one embodiment, authentication process may usemultiple gaze sequences 220 from multiple users 110 (coupled with faceprints) to provide an action. For example, access to a secure area mayonly be granted based on two people providing separate gazesequences/facial recognition together (or in sequence).

In one embodiment, the gaze quadrants 201A-201N could each have at leastone camera 101 in each gaze quadrant 201. In this embodiment, the gazesequence 220 is detected when the user 110 is directly looking at theparticular camera 101 in the gaze quadrant 201. For example, the gazequadrants 201A-201N could be shown on an optical input screen(essentially an array of cameras 101 that captures gaze over the gazequadrants 201A-201N of the optical input screen). The optical inputscreen captures the direct gaze of the user 110 for each gaze quadrant201A-201N. In one embodiment, the cameras 101 in each gaze quadrant 201may be in various places. For example, the cameras 101 may be two ormore cameras 101 attached to the top of monitor, embedded in the top ofthe monitor, placed around the monitor (e.g., one on the top, rightside, left side, and bottom), around a room, around a doorway, by a doorframe, and/or the like.

In the embodiments that use the gaze sequence 220 for authenticating auser 110, the gaze application 105 could use additional authenticationmetrics, such as, a password, a digital certificate, a biometric (e.g.,a fingerprint scan, an iris (eye) scan, a palm print scan, a hearingrange test, and/or the like) to authenticate the user 110.

The gaze application 105 may use different time periods to detect thegaze sequence 220. For example, there may be a two second gaze periodwhere the user 110 looks at the specific gaze quadrant 201. When theuser 110 is done with the gaze sequence 220, the user 110 can look for alonger period of time (e.g., 4 seconds) on the last gaze quadrant 201 toend the gaze sequence 220 entry (or look away for a period of time).

FIG. 3 is a flow diagram of a process for using gaze to control anelectrical and/or mechanical element 104. Illustratively, the camera(s)101 and the gaze application 105 are stored-program-controlled entities,such as a computer or microprocessor 102, which performs the method ofFIGS. 3-5 and the processes described herein by executing programinstructions stored in a computer readable storage medium, such as amemory 103 (i.e., a computer memory, a hard disk, and/or the like).Although the methods described in FIGS. 3-5 are shown in a specificorder, one of skill in the art would recognize that the steps in FIGS.3-5 may be implemented in different orders and/or be implemented in amulti-threaded environment. Moreover, various steps may be omitted oradded based on implementation.

The process starts in step 300. The gaze application 105 determines, instep 302, if a gaze interaction is detected. The gaze interaction iswith one or more of the cameras 101. The gaze interaction can be wherethe user 110 is looking directly at the camera 101 or at an angle (e.g.,as described in FIG. 2) from directly looking at the camera 101. If agaze interaction is not detected in step 302, the gaze application 105determines, in step 308, if the process is complete. If the process iscomplete in step 308, the process ends in step 310. Otherwise, theprocess goes back to step 302.

If the gaze interaction is detected in step 302, the gaze application105 determines, in step 304, if there is an action associated with thegaze interaction. If there is not an action associated with the gazeinteraction in step 304, the process goes to step 308. Otherwise, if thegaze application 105 identifies the associated action for controllingthe electrical and/or mechanical element 104 in step 304, the gazeapplication 105 the initiates the action for controlling the electricaland/or mechanical element 104 in step 306. The process then goes to step308.

To further illustrate the process of FIG. 3, consider the followingnon-limiting examples. While the non-limiting examples are discussed inregard to FIG. 3, one of skill in the art would clearly understand thatthese illustrative examples will also work with the embodimentsdiscusses in FIGS. 1, 2, 4, and 5.

In one embodiment, assume that the electrical and/or mechanicalelement(s) 104 is a listening device with a camera 101 (e.g., an Amazon®Alexa). When the user 110 gazes at the camera 101 in the listeningdevice, the listening device is turned on and asks the user 110 ifhe/she wants to place an order or asks a question. The advantage to thisapproach is that the listening device is only active for the period oftime when the user 110 want to make an order or ask a question. Thelistening device is not constantly listening, thus addressing privacyconcerns. In a similar manner, the user 110 can gaze at the listeningdevice for a period of time to turn off the listening device (or basedon a time period where no input is received).

The gaze detection of FIG. 3 can be used in a variety ways in a varietyof devices/systems, such as, opening/closing a garage door, turningon/off a specific light or group of lights, turning on/off a specificsprinkler or group of sprinklers, alarming an alarm system, changing atemperature setting of a heating/air system, turning on/off theheating/air system, opening/closing a door, securing a house, activatinga computer, initiating a call, turning on/off a printer/scanner, and/orthe like.

In one embodiment, the gaze detection can be coupled with other types ofuser input, such as gestures, voice, touch, and/or the like. Forexample, a heating/air system may be turned on by the user 110 gazing ata controller for the heating/air system. The user 110 then may turn thetemperature up or down based on a hand gesture. For example, the user110 may turn down the heat setting by moving their hand in a downwardmotion or turn up the heat setting by moving their hand in an upwardmotion. In some embodiments, both gestures and gaze events may berequired to initiate an action. For example, the gaze coupled with theuser 110 moving their hand up may be required to turn the hearing systemtemperature setting up. The gaze/gesture events may also use other typesof input, such as voice and touch.

Alternatively, the gaze detection can be coupled with touch and/or voiceinput. For example, a user's gaze may turn on a device and then the user110 can press a button, touch a touch screen, speak a verbal command,and/or the like to implement a specific action.

FIG. 4 is a flow diagram of a process for using a gaze sequence 220. Theprocess of FIG. 4 goes between steps 304 and 306 of FIG. 3. The processof FIG. 4 is based on the discussion of the gaze sequence 220 describedin FIG. 2. As discussed in FIG. 2, the gaze sequence 220 is a series ofgaze interactions that are typically used as part of a userauthentication process.

However, in one embodiment, the gaze sequence 220 may be used for anon-authentication actions, such as multi-function control process. Forexample, a first gaze sequence 220 may be used to turn on/off a firstlight set and a second gaze sequence 220 may be used to turn on/off asecond light set.

The gaze detection in step 302 of FIG. 2 may switch modes based oncontext. For example, when the user 110 first accesses a system, thegaze interaction of step 302 may be for detecting a gaze sequence 220for authenticating a user 110. After the user 110 is authenticated, thegaze interaction of step 302 may switch to detecting a different type ofgaze interaction. For example, the gaze interaction may cause a lightingsystem to turn on/off.

After determining that there is an action associated with the detectedgaze interaction in step 304, the gaze application 105 determines if thegaze interaction is supposed to be for a gaze sequence 220 in step 400.If the gaze application 105 is not looking for a gaze sequence 220 instep 400, the process goes to step 306. Otherwise, if the gazeapplication 105 is looking for a gaze sequence 220 in step 400, the gazeapplication 105 determines, in step 402, if the entered gaze sequence220 matches a stored gaze sequence (e.g., a stored PIN). If the gazesequence matches, in step 402, the process goes to step 306 andinitiates the action (e.g., to authenticate the user 110 and grantaccess to a secure room). If the gaze sequence 220 does not match instep 402, the gaze application 105 may optionally notify the user 110 ofthe invalid gaze sequence 220 in step 404. For example, the system maysay “invalid gaze sequence entered. Try again.” The process then goes tostep 308.

FIG. 5 is a flow diagram of a process for identifying a gaze quadrant201. FIG. 5 is an exemplary embodiment of step 306 of FIG. 3 wheremultiple gaze quadrants 201 are used (versus only a single gaze quadrant201). After determining that there is an action associated with the gazeinteraction in step 304, the gaze application 105 identifies the gazequadrant 201 that the user 110 is looking at in step 500. The gazeapplication 105 identifies the action based on the gaze quadrant 201 forcontrolling the electrical and/or mechanical element 104 in step 502.The process then goes to step 308.

Any of the steps, functions, and operations discussed herein can beperformed continuously and automatically.

To avoid unnecessarily obscuring the present disclosure, the precedingdescription omits a number of known structures and devices. Thisomission is not to be construed as a limitation of the scope of theclaimed disclosure. Specific details are set forth to provide anunderstanding of the present disclosure. It should, however, beappreciated that the present disclosure may be practiced in a variety ofways beyond the specific detail set forth herein.

Furthermore, while the exemplary embodiments illustrated herein show thevarious components of the system collocated, certain components of thesystem can be located remotely, at distant portions of a distributednetwork, such as a LAN and/or the Internet, or within a dedicatedsystem. Thus, it should be appreciated, that the components of thesystem can be combined into one or more devices, such as a server,communication device, or collocated on a particular node of adistributed network, such as an analog and/or digital telecommunicationsnetwork, a packet-switched network, or a circuit-switched network. Itwill be appreciated from the preceding description, and for reasons ofcomputational efficiency, that the components of the system can bearranged at any location within a distributed network of componentswithout affecting the operation of the system.

Furthermore, it should be appreciated that the various links connectingthe elements can be wired or wireless links, or any combination thereof,or any other known or later developed element(s) that is capable ofsupplying and/or communicating data to and from the connected elements.These wired or wireless links can also be secure links and may becapable of communicating encrypted information. Transmission media usedas links, for example, can be any suitable carrier for electricalsignals, including coaxial cables, copper wire, and fiber optics, andmay take the form of acoustic or light waves, such as those generatedduring radio-wave and infra-red data communications.

While the flowcharts have been discussed and illustrated in relation toa particular sequence of events, it should be appreciated that changes,additions, and omissions to this sequence can occur without materiallyaffecting the operation of the disclosed embodiments, configuration, andaspects.

A number of variations and modifications of the disclosure can be used.It would be possible to provide for some features of the disclosurewithout providing others.

In yet another embodiment, the systems and methods of this disclosurecan be implemented in conjunction with a special purpose computer, aprogrammed microprocessor 102 or microcontroller and peripheralintegrated circuit element(s), an ASIC or other integrated circuit, adigital signal processor, a hard-wired electronic or logic circuit suchas discrete element circuit, a programmable logic device or gate arraysuch as PLD, PLA, FPGA, PAL, special purpose computer, any comparablemeans, or the like. In general, any device(s) or means capable ofimplementing the methodology illustrated herein can be used to implementthe various aspects of this disclosure. Exemplary hardware that can beused for the present disclosure includes computers, handheld devices,telephones (e.g., cellular, Internet enabled, digital, analog, hybrids,and others), and other hardware known in the art. Some of these devicesinclude processors (e.g., a single or multiple microprocessors 102),memory 103, nonvolatile storage, input devices, and output devices.Furthermore, alternative software implementations including, but notlimited to, distributed processing or component/object distributedprocessing, parallel processing, or virtual machine processing can alsobe constructed to implement the methods described herein.

In yet another embodiment, the disclosed methods may be readilyimplemented in conjunction with software using object or object-orientedsoftware development environments that provide portable source code thatcan be used on a variety of computer or workstation platforms.Alternatively, the disclosed system may be implemented partially orfully in hardware using standard logic circuits or VLSI design. Whethersoftware or hardware is used to implement the systems in accordance withthis disclosure is dependent on the speed and/or efficiency requirementsof the system, the particular function, and the particular software orhardware systems or microprocessor 102 or microcomputer systems beingutilized.

In yet another embodiment, the disclosed methods may be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory 103, a special purpose computer, a microprocessor102, or the like. In these instances, the systems and methods of thisdisclosure can be implemented as a program embedded on a personalcomputer such as an applet, JAVA® or CGI script, as a resource residingon a server or computer workstation, as a routine embedded in adedicated measurement system, system component, or the like. The systemcan also be implemented by physically incorporating the system and/ormethod into a software and/or hardware system.

Although the present disclosure describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Other similar standards and protocols not mentioned hereinare in existence and are considered to be included in the presentdisclosure. Moreover, the standards and protocols mentioned herein, andother similar standards and protocols not mentioned herein, areperiodically superseded by faster or more effective equivalents havingessentially the same functions. Such replacement standards and protocolshaving the same functions are considered equivalents included in thepresent disclosure.

The present disclosure, in various embodiments, configurations, andaspects, includes components, methods, processes, systems and/orapparatus substantially as depicted and described herein, includingvarious embodiments, sub-combinations, and subsets thereof. Those ofskill in the art will understand how to make and use the systems andmethods disclosed herein after understanding the present disclosure. Thepresent disclosure, in various embodiments, configurations, and aspects,includes providing devices and processes in the absence of items notdepicted and/or described herein or in various embodiments,configurations, or aspects hereof, including in the absence of suchitems as may have been used in previous devices or processes, e.g., forimproving performance, achieving ease, and/or reducing cost ofimplementation.

The foregoing discussion of the disclosure has been presented forpurposes of illustration and description. The foregoing is not intendedto limit the disclosure to the form or forms disclosed herein. In theforegoing Detailed Description for example, various features of thedisclosure are grouped together in one or more embodiments,configurations, or aspects for the purpose of streamlining thedisclosure. The features of the embodiments, configurations, or aspectsof the disclosure may be combined in alternate embodiments,configurations, or aspects other than those discussed above. This methodof disclosure is not to be interpreted as reflecting an intention thatthe claimed disclosure requires more features than are expressly recitedin each claim. Rather, as the following claims reflect, inventiveaspects lie in less than all features of a single foregoing disclosedembodiment, configuration, or aspect. Thus, the following claims arehereby incorporated into this Detailed Description, with each claimstanding on its own as a separate preferred embodiment of thedisclosure.

Moreover, though the description of the disclosure has includeddescription of one or more embodiments, configurations, or aspects andcertain variations and modifications, other variations, combinations,and modifications are within the scope of the disclosure, e.g., as maybe within the skill and knowledge of those in the art, afterunderstanding the present disclosure. It is intended to obtain rights,which include alternative embodiments, configurations, or aspects to theextent permitted, including alternate, interchangeable and/or equivalentstructures, functions, ranges, or steps to those claimed, whether or notsuch alternate, interchangeable and/or equivalent structures, functions,ranges, or steps are disclosed herein, and without intending to publiclydedicate any patentable subject matter.

Embodiments include a device comprising: a microprocessor; a computerreadable medium, coupled with the microprocessor and comprisingmicroprocessor readable and executable instructions that program themicroprocessor to: detect a gaze interaction from a user or animal basedon information received from a camera; in response to detecting the gazeinteraction, identify an action for controlling one or more of anelectrical element and a mechanical element; and initiate the action tocontrol the one or more of the electrical element and the mechanicalelement.

Aspects of the above device include wherein the camera is in the device,wherein the action is to power on the device, and wherein themicroprocessor readable and executable instructions further program themicroprocessor to ask if the user wants to speak a voice command thatcontrols the device.

Aspects of the above device include wherein the gaze interaction isbased on gaze information received from a plurality of different camerasand wherein the gaze interaction is a sequence of gaze interactions withthe plurality of different cameras that is used for at least one of: auser authentication process and a multi-function control process.

Aspects of the above device include wherein the sequence of gazeinteractions with the plurality of different cameras is used for theuser authentication process.

Aspects of the above device include wherein the sequence of gazeinteractions with the plurality of different cameras is used for themulti-function control process.

Aspects of the above device include wherein the gaze interactioncomprises a plurality of gaze quadrants that are used to initiate aplurality of different actions to control the one or more of theelectrical element and the mechanical element.

Aspects of the above device include wherein the plurality of gazequadrants are used as part of a gaze sequence for authenticating a user.

Aspects of the above device include wherein the gaze interaction iscoupled with one or more gestures in order to initiate the action tocontrol the one or more of the electrical element and the mechanicalelement.

Aspects of the above device include wherein detecting the gazeinteraction is used to control an alarm system and wherein the action isto initiate different alarm actions based on whether the detected gazeinteraction is a direct gaze interaction or an indirect gazeinteraction.

Aspects of the above device include wherein the gaze interaction iscoupled with one or more touch events to initiate the action to controlthe one or more of the electrical element and the mechanical element.

Aspects of the above device include wherein the gaze interaction iscoupled with one or more voice commands to initiate the action tocontrol the one or more of the electrical element and the mechanicalelement.

Embodiments include a method comprising: detecting, by a microprocessor,a gaze interaction from a user or animal based on information receivedfrom a camera; in response to detecting the gaze interaction,identifying, by the microprocessor, an action for controlling one ormore of an electrical element and a mechanical element; and initiating,by the microprocessor, the action to control the one or more of theelectrical element and the mechanical element.

Aspects of the above method include wherein the camera is in the device,wherein the action is to power on the device, and further comprising,asking, by the microprocessor, if the user wants to speak a voicecommand that controls the device.

Aspects of the above method include wherein the gaze interaction isbased on gaze information received from a plurality of different camerasand wherein the gaze interaction is a sequence of gaze interactions withthe plurality of different cameras that is used for at least one of: auser authentication process and a multi-function control process.

Aspects of the above method include wherein the sequence of gazeinteractions with the plurality of different cameras is used for theuser authentication process.

Aspects of the above method include wherein the gaze interactioncomprises a plurality of gaze quadrants that are used to initiate aplurality of different actions to control the one or more of theelectrical element and the mechanical element.

Aspects of the above method include wherein the plurality of gazequadrants are used as part of a gaze sequence for authenticating a user.

Aspects of the above method include wherein the gaze interaction iscoupled with one or more gestures in order to initiate the action tocontrol the one or more of the electrical element and the mechanicalelement.

Aspects of the above method include wherein detecting the gazeinteraction is used to control an alarm system and wherein the action isto initiate different alarm actions based on whether the detected gazeinteraction is a direct gaze interaction or an indirect gazeinteraction.

Aspects of the above method include wherein the gaze interaction iscoupled with one or more touch events to initiate the action to controlthe one or more of the electrical element and the mechanical element.

The phrases “at least one,” “one or more,” “or,” and “and/or” areopen-ended expressions that are both conjunctive and disjunctive inoperation. For example, each of the expressions “at least one of A, Band C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “oneor more of A, B, or C,” “A, B, and/or C,” and “A, B, or C” means Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more,” and “at least one” can beused interchangeably herein. It is also to be noted that the terms“comprising,” “including,” and “having” can be used interchangeably.

The term “automatic” and variations thereof, as used herein, refers toany process or operation, which is typically continuous orsemi-continuous, done without material human input when the process oroperation is performed. However, a process or operation can beautomatic, even though performance of the process or operation usesmaterial or immaterial human input, if the input is received beforeperformance of the process or operation. Human input is deemed to bematerial if such input influences how the process or operation will beperformed. Human input that consents to the performance of the processor operation is not deemed to be “material.”

Aspects of the present disclosure may take the form of an embodimentthat is entirely hardware, an embodiment that is entirely software(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module,” or “system.”Any combination of one or more computer-readable medium(s) may beutilized. The computer-readable medium may be a computer-readable signalmedium or a computer-readable storage medium.

A computer-readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer-readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer-readable signal medium may be any computer-readable medium thatis not a computer-readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer-readable medium may be transmitted using anyappropriate medium, including, but not limited to, wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

The terms “determine,” “calculate,” “compute,” and variations thereof,as used herein, are used interchangeably and include any type ofmethodology, process, mathematical operation or technique.

What is claimed is:
 1. A device comprising: a microprocessor; a computerreadable medium, coupled with the microprocessor and comprisingmicroprocessor readable and executable instructions that program themicroprocessor to: detect a gaze interaction sequence from a user oranimal based on information received from a camera, wherein the gazeinteraction sequence comprises multiple gaze interactions in multiplelocations; in response to detecting the gaze interaction sequence,identify an action for controlling one or more of an electrical elementand a mechanical element; and initiate the identified action to controlthe one or more of the electrical element and the mechanical element. 2.The device of claim 1, wherein the camera is in the device, wherein theaction is to ask if the user wants to speak a voice command thatcontrols the device.
 3. The device of claim 1, wherein the action is atleast one of: a user authentication process and a multi-function controlprocess.
 4. The device of claim 3, wherein the user authenticationprocess comprises the user entering a Personal Identification Number(PIN).
 5. The device of claim 3, wherein the multi-function controlprocess comprises at least a first action to control a first electricalelement or mechanical element and a second action to control a secondelectrical element or mechanical element.
 6. The device of claim 1,further comprising: a plurality of gaze quadrants that are used todetect the multiple gaze interactions in the multiple locations.
 7. Thedevice of claim 6, wherein the plurality of gaze quadrants are used aspart of the gaze interaction sequence for authenticating the user. 8.The device of claim 1, wherein the gaze interaction sequence is coupledwith one or more gestures in order to initiate the action to control theone or more of the electrical element and the mechanical element.
 9. Thedevice of claim 1, wherein detecting the gaze interaction sequence isused to control an alarm system and wherein the action is to initiatedifferent alarm actions based on the detected gaze interaction sequence.10. The device of claim 1, wherein the gaze interaction sequence iscoupled with one or more touch events to initiate the action to controlthe one or more of the electrical element and the mechanical element.11. The device of claim 1, wherein the gaze interaction sequence iscoupled with one or more voice commands to initiate the action tocontrol the one or more of the electrical element and the mechanicalelement.
 12. A method comprising: detecting, by a microprocessor, a gazeinteraction from a user or animal based on information received from acamera, wherein the gaze interaction is determined using a plurality ofgaze quadrants, and wherein each of the plurality of gaze quadrants isassociated with a different action; in response to detecting the gazeinteraction, identifying, by the microprocessor, an action forcontrolling one or more of an electrical element and a mechanicalelement; and initiating, by the microprocessor, the identified action tocontrol the one or more of the electrical element and the mechanicalelement.
 13. The method of claim 12, wherein the camera is in a device,wherein the action is asking, by the microprocessor, if the user wantsto speak a voice command that controls the device.
 14. The method ofclaim 12, wherein the gaze interaction is based on a sequence of gazeinteractions, wherein the sequence of gaze interactions comprisemultiple gaze interactions in the plurality of gaze quadrants.
 15. Themethod of claim 14, wherein the sequence of gaze interactions is usedfor a user authentication process.
 16. The method of claim 14, whereinthe sequence of gaze interactions is used to initiate a plurality ofdifferent actions to control the one or more of the electrical elementand the mechanical element.
 17. The method of claim 12, wherein the gazeinteraction is used for a multi-function control process.
 18. The methodof claim 12, wherein the gaze interaction is coupled with one or moregestures in order to initiate the action to control the one or more ofthe electrical element and the mechanical element.
 19. The method ofclaim 12, wherein detecting the gaze interaction is used to control analarm system and wherein the action is to initiate different alarmactions based on the detected gaze interaction.
 20. The method of claim12, wherein the gaze interaction is coupled with one or more touchevents to initiate the action to control the one or more of theelectrical element and the mechanical element.